df.columns comes with overhead unfortunately, I think we should avoid calling it where possible. How much overhead depends on the operation

I was hoping we could do something like we do for Polars. That is to say, when we do select / with_columns, we wrap them in try/except, and in the except block we intercept the error message to give a more useful / unified one

Ah interesting, I was not aware 😕

What is happening in the background in duckdb that causes this overhead ? Do you have a link to the docs? (Just want to learn more)

Also, is it a specific caveat of duckdb? I don't think we should worry about that in spark-like but I might be wrong

I will update the code tonight anyway (but of course feel free to add commits to this branch if you need it for today's release)

df.columns comes with overhead unfortunately, I think we should avoid calling it where possible. How much overhead depends on the operation

@MarcoGorelli could we add that to (#805) and put more of a focus towards it? 🙏

I don't think it's documented, but evaluating .columns may sometimes require doing a full scan. Example:

In [48]: df = pl.DataFrame({'a': rng.integers(0, 10_000, 100_000_000), 'b': rng.integers(0, 10_000, 100_000_000)})

In [49]: rel = duckdb.table('df')
100% ▕████████████████████████████████████████████████████████████▏

In [50]: rel1 = duckdb.sql("""pivot rel on a""")

In [51]: %timeit rel.columns
385 ns ± 7.62 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [52]: %timeit rel1.columns
585 μs ± 3.8 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Granted, we don't have pivot in the Narwhals lazy API, but a pivot may appear in the history of the relation which someone passes to nw.from_native, and the output schema of pivot is value-dependent (😩 )

The same consideration should apply to spark-like

How do those timings compare to other operations/metadata lookups on the same tables?

.alias for example is completely non-value-dependent, so that stays fast

In [60]: %timeit rel.alias
342 ns ± 2.3 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [61]: %timeit rel1.alias
393 ns ± 2.6 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

Not 100% sure about this error message. I don't we can access the missing column names at this level, am I missing something?

I think what you're written is great - even though we can't access them, we can still try to be helpful

I split the test into lazy and eager to simplify a bit the if-else statements. I hope it is a bit more readable ?

Before it was msg = "e|f". Now it is a bit stricter

before this was not tested for polars

sqlframe and pystpark.connect raise errors at collect. 😕

I need to double check pystpark.connect. Currently cannot set it up locally... Working on it ⏳

Do you have an idea on how to deal with these?

drop should be already tested in drop_test.py

We use parse_columns_to_drop and therefore raise our ColumnNotFoundError.from_missing_and_available_column_names(missing_columns=missing_columns, available_columns=cols) for every backend